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ABSTRACT 
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Abstract 

Many researchers are attempting to develop automated 
instructional development systems to guide subject matter experts 
through the lengthy and difficult process of courseware 
development. Because the targeted users often lack instructional 
design expertise, a great deal of emphasis has been placed on the 
use of artificial intelligence (AI) to incorporate expert 
instructional design knowledge in these automated systems 
(Duchastel, 1990) . This paper presents a taxonomy describing 
various uses of AI techniques in automated instructional 
developments systems. In addition, two specific systems being 
developed at the Air Force Armstrong Laboratory (AIDA and GAIDA) 
are reviewed. The initial formative evaluation of GAIDA is also 
reported. We conclude with some remarks about prospects for the 
future use of AI in automated instructional development systems. 



Background 

Several Department of Defense agencies, educational 
corporations, and academic institutions are attempting to 
automate part or all of the Instructional Systems Development 
(ISD) process in an attempt to improve the productivity and 
effectiveness of novice training developers in designing 
interactive computer-based instruction (CBI) courseware 
(Duchastel, 1990). Use of expert systems, intelligent lesson 
templates, front-end analysis tools, and sophisticated CBI 
authoring environments are currently being designed, developed, 
and tested. 

Typically, the designer of instruction in the Air Force is a 
subject-matter expert who has completed a brief course on 
designing instruction (Spector, Muraida, & Dallman, 1990) . For 
example, he or she may be a staff sergeant who is experienced in 
some particular aspects of aircraft maintenance. With a small 
amount of training in instructional design, this person is asked 
to bring equipment-system knowledge to bear in the design of 
training for the maintenance of highly complex technical 
equipment. 

In response to this situation, the Air Force Armstrong 
Laboratory (Human Resources Directorate) has an exploratory 
research and development effort in the area of automated 
instructional design called the Advanced Instructional Design 
Advisor (AIDA) . AIDA will provide an intelligent, rule-based 
framework and a powerful collection of object-oriented authoring 

1 



3 



INTELLIGENT FRAMEWORKS FOR INSTRUCTIONAL DESIGN 

tools to assist the subject-matter expert in developing effective 
courseware. AIDA derives its primary theoretical basis from 
transaction theory (Merrill, Li, & Jones, 1990), which proposes 
executable and configurable frameworks (called transaction 
shells) for particular kinds of lesson objectives. 

A complementary and parallel effort, the Guided Approach for 
Instructional Design Advising (GAIDA) , is also being developed at 
Armstrong Laboratory. Robert M. Gagne, who has been serving as a 
National Research Council Senior Associate, is supervising the 
GAIDA project. GAIDA has goals similar to those of AIDA (provide 
a powerful and intelligent framework for instructional design) , 
but it approaches those goals from the perspective of the human 
designer. 

AIDA places a great deal of the burden for intelligent 
design on the design system. AIDA*s intelligence appears in the 
form of a rule-based instructional transaction configuration 
system and in the form of default values for particular 
transaction shells that are appropriate for a specified type of 
lesson objective (Merrill, Li, & Jones, 1990). 

The GAIDA system aims to provide guidance to Air Force 
instructional designers that follows the approximate sequence 
described by Gagne, Briggs, & Wager (1992) as the nine events of 
instruction . As indicated by these authors, the particular form 
of these nine events will be influenced by the intended 
capability to be learned, or in other words, by the particular 
kind of learning outcome that is expected. 

Both GAIDA and AIDA share the view that instruction should 
be aimed at integrated and purposeful human activities, or 
enterprises (Gagne & Merrill, 1990) . It is likely that the two 
systems will be integrated in a manner that allows GAIDA to serve 
in a stand-alone capacity to instructional designers or to 
provide on-line examples and guidelines to users of AIDA. 



Intelligent Instructional Design 

Intelligence 

In order to provide a context for the two systems described 
below, we need a working definition of artificial intelligence 
(AI) , There are, of course, many definitions of AI in the 
literature. Some emphasize the psychological aspects of human 
intelligence and various methods for modeling those processes. 
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Others emphasize the mathematical complexity of certain problems 
addressed by computer scientists. A more neutral approach is 
taken by Rich and Knight who define artificial intelligence to be 
"the study of how to make computers do things which, at the 
moment, people do better." (1991, p. 3) 

We propose this working definition of AI: Artificial 
intell.igence refers to those aspects of computer science and 
engineering which are used in systems that most practitioners 
regard as intelligent systems. Our definition is intentionally 
weak and circular. It is not clear what benefits are to be 
derived from endless debates concerning the so-called 
intelligence of certain machines and computer programs. Another 
way of avoiding those debates is to use 'engineered cognition' in 
place of 'artificial intelligence'. [This is Gagne's suggestion 
for avoiding unnecessary difficulties on thi«5 issue.] 

Moreover, AI is likely to make progress and change 
significantly over time, and our definition recognizes this basic 
fact concerning the emergence and evolution of AI. The 
techniques of computer science which are generally regarded as 
intelligent include those found in artificial neural networks, 
case-based systems, diagnostic systems, dialogue managers, expert 
systems, natural language processors, planning architectures, 
robotic systems, rule-based systems, semantic networks, and so 
on. 

Instructional Systems Development fiSD) 

The next element of our context pertains to the process of 
developing instruction. The most prevalent models of 
instructional development are those based on an engineering 
approach to curriculum and are called ISO models (Andrews and 
Goodson, 1980; Gagne, Tennyson, & Gettman, 1991). These models 
typically divide the process of developing instruction into five 
stages or phases (See Table 1 below) . Older ISD models often 
fail to account for relevant cognitive aspects of the learning 
task. For example, a behavioral ISD model for performing task 
analysis might lead the instructional designer to describe 
particular procedures carried out by the troubleshooter without 
any mention of a mental model that might have influenced what was 
guiding a troubleshooter through a maze of subprocedures . 
Tennyson (Gagne, Tennyson, & Gettman, 1991) has argued that ISD 
should be updated to include relevant principles derived from 
cognitive psychology. 

In addition, Tennyson argues that evaluation belongs in each 
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of the phases and that formative evaluation is an on-going 
process. As a consequence, for our purposes in this pmxer we 
propose the ISD model in Table 1 (below) . 



Table 1 



Typical ISD Model 



ISD PHASE 


TYPICAL GOALS 


Analysis 


Define training requirements. 




Analyze target populations. 




Establish performance levels. 


Design 


Specify instructional objectives. 


Group and sequence objectives. 




Design instructional treatments. 


Production 


Develop learning activities. 




Develop test items. 




Evaluate prototypes. 


Implementat ion 


Implement learning activities. 




Administer test items. 




Assess student results. 


Maintenance 


Revise course materials. 




Revise test items. 




Assess course effectiveness. 



Table 1 should not be interpreted in a rigid manner. The 
purpose of this table is to provide a framework for identifying 
where AI techniques may have an application in the instructional 
development process. We mean only to suggest that the kinds of 
goals identified in the second column are typically assocr'ated 
with the ISD phase in the first column. We believe that the ISD 
process is iterative and cyclic by nature, that evaluations occur 
within each phase, and that the phases are intmxtwined and may 
not be accomplished in a strictly linear fashion. 
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Intelligent Instructional Development 

Having provided a working definition of AI and an ISD model, 
we are now in a position to provide a framework for incorporating 
AI into instructional design and to locate the area of our own 
research efforts. Table 2 provides a tentative taxonomy for the 
various ways that AI has been incorporated into the ISD process. 
We have included in each phase examples of AI efforts that have 
been developed or are at least in the design stage. The focus of 
our own efforts is in the instructional design phase, although 
AIDA does provide assistance with production and implementation. 

These examples of applicable AI techniques in the ISD domain 
are not meant to be exhaustive. What we offer in Table 2 is a 
way to classify various ei forts to develop intelligent 
instructional systems. 

It is worth noticing that no examples of intelligent 
applications in the maintenance phase are identified. However, 
it is possible to imagine an automated instructional system that 
monitored either the instructional development process or the 
progress of learners using the system, processed the results, and 
filtered those results through a set of rules which prescribed 
certain types of modifications when particular kinds of results 
were noticed observed. 

For exarple, a system could record and analyze answers to 
questions. Jf a particular question was never answered 
correctly, the system might recommend a remedy for the situation. 
A system might also monitor where learners spent most of their 
time with the system and analyze how that time contributed to 
learning outcomes. If it appeared that time was wasted in one 
part of the system, then the system might recommend some kind of 
remedy for that situation. 

There is nothing in this schema to prevent a particular 
systom from being categorized in more than one area. In fact, we 
view AIDA as an intelligent application in the middle three 
phases of ISD, although our research interests are clearly 
focused on the design phase. 
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Table 2 

Taxonomy of AI Techniajes in the ISP Process 



ISP PROCESS 
Analysis 



Design 



Production 



Implementation 



APPLICABLE AI TECHNIQUES 

Intelligent Training Requirements Tools 
Cognitive Task Analysis 

(e.g., GOMS, PARI) 
Pecision Support Systems 

(e.g. , TPS) 

Instructional Pesign Advisors 

On-line Examples & Guidelines 

(e.g., GAIPA) 
Rule- & Case-based Guidance 

(e.g., AIPA) 
Intelligent Tutoring and 

Critiquing Systems 

Intelligent Pevelopment Tools 

Mini-advisors for Graphics, 
Audio, Video, and 
Interface Issues 
Intelligent Lesson Templates 
(e.g., AIPA) 

Adaptive Pelivery Systems 

Intelligent Tutoring Systems 
Adaptive Testing Techniques 
Non-intelligent Tutors 
(e.g., AIPA) 



Maintenance 



Monitoring & Feedback of Results 



As already mentioned the two projects which are described 
below focus on the application of AI to the instructional design 
phase. It is worth noting that GAIPA and AIPA represent only two 
of a number of possible approaches to intelligent instructional 
design. Tenny on (1991) has proposed the more ambitious approach 
of building an xntelligent tutoring system (ITS) for the domain 
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of instructional design. Duchastel (1990) has proposed an expert 
critiquing system which can evaluate designs created by 
relatively experienced instructional designers. It is our 
belief, however, that is premature to proceed with such ambitious 
efforts. Indeed, the validated results of efforts like GAIDA and 
AIDA will form an essential part of an expert ISD critiquer or an 
ITS for ISD. 



AIDA 

Objectives of AIDA 

The primary objective of AIDA is to encapsulate human 
knowledge about learning and instruction pertinent to electronics 
maintenance training tasks in an intuitive system accessible to 
Air Force training specialists. The motivation for this effort 
is the projected increase in demand for computer-based 
instruction coupled with declining training development budgets 
and a scarcity of courseware design experts in the Air Force. In 
short, the Air Force must use subject matter experts to design, 
develop, and deliver computer-based course materials. 

This situation (using subject matter experts to design and 
develop instruction) is more tolerable with regard to classroom 
instruction, and it is the de facto norm in our society. 
However, what works in the classroom may not work well in a 
computer-based setting. Computers do not respond well to puzzled 
looks and bored faces. Great care must be taken in planning and 
implementing an effective computer-based learning environment. 
The challenge for the Air Force is to do the requisite careful 
planning with the talent on hand. 

Theoretical Framework for AIDA 

The theoretical framework consists primarily of Merrill's 
second generation instructional design theory (Merrill, Li, & 
Jones, 1990). Merrill's theory grows out of numerous 
inadequacies with previous instructional theories, including his 
own Component Display Theory, which failed to account for the 
integrated nature of learning tasks and the unique capabilities 
of computers to support specific learning objectives. 

Second generation instructional design theory is built 
around integrated human performances called enterprises (Gagne & 
Merrill, 1990), which can be decomposed into various entities 
(abstract or concrete objects) , activities (which involve 
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humans) , and processes (which proceed without human involvement) . 
Merrill postulates specific kinds of instruction for each kind of 
object. For example, entities typically have named parts so 
paired associate learning can be encapsulated in a pre-designed 
framework (transaction shell) to initiate teaching about 
entities. 

Human expertise about learning and instruction is being 
captured in a collection of transaction shells that are 
appropriate to electronics maintenance training. Additional 
human expertise will be captured in a front-end advisor which 
will interact with users (subject matter experts) to collect 
domain knowledge and information about the training setting and 
then configure an initial (but alterable) set of transactions for 
various lesson objectives. 

Methodology 

The methodology has been tc pursue an incremental 
development of AIDA. The first phase included a needs assessment 
performed at an Air Force technical training center and an 
intense analysis of this challenging problem by seven noted 
educational researchers (Robert Gagne, Henry Halff, David 
Merrill, Harry O'Neil, Martha Poison, Charles Reigeluth, and 
Robert Tennyson) and numerous military advisors (including Jerry 
Barucky, Brian Dallman, John Ellis, Mary Marlino, Milt Nielsen, 
Rich Ranker, Bob Seidel, and Richard Thurman) . The outcome of 
the first phase was a requirements analysis and a conceptual 
framework for AIDA (Spector, Muraida, & Dallman, 1990) . 

The second phase made use of the same academic and military 
consultants and focused on developing a refined set of functional 
requirements and conducting field tests of an initial authoring 
environment provided by Merrill. Also included in the second 
phase was a technology assessment of second generation 
instructional design theory (Canfield & Spector, 1991) . The 
outcome of the second phase was a more detailed conceptual 
framework and a set of design specifications for an experimental 
system (XAIDA) . 

The project is currently in its third phase, which is an 
implementation and evaluation of the XAIDA. The software design 
and development is being performed by Mei Technology, Inc. 
Instructional design expertise is being provided on an on-going 
basis by Merrill and Halff. The system is being coded in an 
object-oriented environment (C++) and implemented on a DeskTop 
III (Intel 80386-based microcomputer) . 

8 
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Current Status of XAIDA 

Field tests of initial transactions for teaching 
nomenclature (entities) and checklist procedures (activities) 
indicated that transaction shells provide an environment that is 
both meaningful and accessible to Air Force training specialists. 
Indications are that productivity can be greatly improved without 
sacrificing courseware quality (Canf ield & Spector, 1991) . XAIDA 
provides a robust environment to test a variety of specific 
instructional prescriptions. As a consequence, XAIDA is an 
excellent tool for conducting research and development that 
pertains to instructional strategies and multimedia 
presentations. 

The first year XAIDA prototype has been completed and an 
evaluation plan set in motion. The evaluation plan begins with 
an internal review, expands the review to include informed 
military and academic advisors, and then evaluates field results 
in Air Force technical training settings. This evaluation will 
continue for two years, and XAIDA will be modified and re-tested 
accordingly. At then of this third phase (1994), a fully 
functional XAIDA will exist. Plans are to conduct additional 
evaluations and validation of the technology, and then to 
transition XAIDA into standard use as AIDA in 1997. 



GAIDA 

Background 

Guided Approach: Instructional Design Advisor (GAIDA) 
represents an approach to the delivery of instruction embodied in 
u computer-based lesson on how to design instruction (Gagne & 
Hansock, 1991; Gagne, Dimitroff, & Whitehead, 1991). GAIDA takes 
the view that if a reasonably intelligent instructional designer 
is provided with meaningful and specific guidance and an 
elaborate and completely worked example that such an individual 
will be capable of constructing effective courseware. 

As described earlier in this article, the design and 
development of ir struct ion follows a five-step model. Details of 
the procedure are described in AFM 50-2, Instructional System 
Development, and in AFP 50-58, Handbook for Designers of 
Instructional Systems. Responsibility for training design in 
maintenai ? specialties is typically given to airmen of non- 
commissioned rank who have job-related experience with the 
equipment on which training is to be given. Instructional 
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designers of this variety receive a small amount of professional 
training, although they may have considerable experience in on- 
the-job maintenance and as Instructors. 

As indicated in a recent report entitled "Revision of the 
Air Force Instructional System Development Process — Baseline 
Analysis Report" (Golas & Shrlver, 1991) , many instructional 
developers find the model described in these publications 
complex, inflexible, and difficult to apply to a variety of 
maintenance jobs. Such a finding suggests a need for 
simplif icatlon of the instructional design process. Since this 
need has actually been recognized for quite some time, a number 
of different suggestions have been put forward as remedies. 
Several of these involve the development of computer-based 
systems such as intelligent tutoring and expert systems, still 
others depend on the use of systematic design procedures that are 
oriented to instructor-led classroom instruction as well as to 
the computer-based variety. 

Training design at the level of the lesson continues to be 
an enterprise that challenges the ingenuity of the Air Force 
designer. When a new weapon system is adopted, or when an 
existing system is modified, the volume of new maintenance 
Information to be taught is often very large. In dealing with 
the necessity of communicating such a mass of information, the 
designer may be sorely tempted to fall into two kinds of error in 
the attempt to simplify his task. These tendencies may be 
described as follows: 

1. Reducing the knowledge to be acquired to the 
declarative form, and thus neglecting the procedural variety of 
knowledge; learning the names of equipment parts is not 
equivalent to learning how to use these parts. 

2. Reducing the Instructional techniques to only two, 
which may be called TELLING and PRACTICE; while these typically 
constitute the core of Instruction, other features of 
instructional strategy, such as elaboration, interactivity, and 
feedback, are often found to enhance instructional effectiveness 
by significant amounts. 

One promising approach to the simplification of the 
instructional design process is automating the procedure of 
design and delivery. An effort to develop and test automation 
techniques is involved in the project AIDA (Hickey, Spector & 
Muraida, 1991) . This project is engaged in the development of 
computer shells representing a number of different instructional 
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strategies, each of which peicains to a different learning goal. 
For example, one goal is identifying equipment and ecmipment 
parts , while another is executing a procedure , and a third is 
interpreting malfunctions . For any one of these goals, the 
shells of instructional procedures can be selected and put 
together so as to represent an el'fective module of instruction 
aimed at that particular goal. The content for the instruction, 
of course, must be selected and added, but how it is presented, 
in a manner conducive to efficient learning, depends upon the 
nature of the shell that is employed. Thus, AIDA uses automated 
computer-human interaction as a means of instruction for 
particular kinds of goals. The goals and the enterprises they 
represent must be identified by the instructional designer. 

Simplification of the process of instructional design can 
also be done in a manner that does not require the degree (or 
kind) of automation involved in the approach of AIDA. 
Instruction, it is evident, consists of a set of events external 
to the learner that occur in a loosely invariant sequence (Gagne, 
1985, 302-319) . These events may be directions or suggestions to 
the learner about what to do next, demonstrations of action 
sequences, pointed references to aspects of the learners' 
environment, reminders of previously learned knowledge, 
solicitation of learner responses, feedback and corrections of 
learner responses, and others (Gagne, 1991) . When a designer of 
instruction follows the prescriptions in this nearly invariant 
sequence (called the "Nine Events of Instruction"; Gagne, Briggs 
& Wager, 1992) further automation is unnecessary. As is true for 
AIDA, the content of these events must be identified and selected 
by the designer. Otherwise, however, a large amount of 
flexibility is possible in the design of specific learner- 
interactive events. 

GAIDA is a project that follows this process of limited 
automation in presenting instruction on designing instruction. 
Because this approach leaves much of the details of design to the 
judgment of the designer, it was particularly appropriate that a 
study aimed at formative evaluation of GAIDA be carried out. 
Results of the initial formative analysis are reported below. 

The GAIDA Project 

Computer-based instruction (CBI) dealing with how to design 
instruction can deliver a set of directions to the novice 
instructional designer. The latter may find these directions 
easy or difficult to understand, easy or difficult to implement. 
For example, if the directions say, in effect, "at this point - 
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tell the learner to recall some previ msly learned knowledge that 
is relevant to the new learning task", the student designer 
should be able to comprehend the message with its inclusion of 
x.ne concepts "this point", "recall", "previously learned", 
"knowledge", and "relevant". The GAIDA approach assumes that 
novice designers are capable of understanding concepts of this 
sort in their precise meanings. The further assumption is made 
that directions of this sort can be "followed", in the sense 
that concrete instances of such an abstraction as "relevant 
knowledge" can be identified and selected from the domain of the 
equipment data base being dealt with. 

GAIDA provides to the novice designer, a set of nine 
directions, in sequence, which are intended to tell the designer 
what kinds of events to devise as instruction. In order, these 
events (Gagne, Briggs & Wager, 1992) are as follows: (1) gain 
attention; (2) describe the goal; (3) stimulate recall of prior 
knowledge; (4) present the material to be learned; (5) give 
learning guidance; (6) elicit performance; (7) provide feedback; 
(8) assess performance; and (9) enhance retention and transfer. 
How readily these directions are understood, and how well they 
are implemented, will determine the auality of the instruction 
that is designed. The need for formative evaluation is therefore 
apparent . 

GAIDA provides printed directions for each of these nine 
events. An example lesson is used to demonstrate the events. In 
the case that was evaluated, the lesson aimed to teach the 32 
steps in the procedure called "Functional Check of the M61A1 
gun", which is the gun in the F-16 aircraft (Gagne, Dimitroff, & 
Whitehead, 1991) . This preventive maintenance procedure is 
carried out with the use of a checklist. Since the system 
components are in two locations, one person executes some steps 
on equipment in the cockpit, while another carries out other 
steps by reaching connectors through panels on the underside of 
the aircraft. Another example pertaining to the training of a 
procedure that must be committed to memory also exists (Gagne & 
Hancock, 1991) . 

The lesson on this procedure is designed to support the 
learning of the following: (1) verbal identification of 
abbreviated names and phrases in the checklist; (2) 
identification of the objects (switches, connectors, etc.) named 
in the checklist; (3) identification of the location or these 
objects; (4) easy progression from each procedural step to the 
next, following the printed checklist. In addition to these 
basics, a few interactive steps are included with the aim of 
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enhancing the "system knowledge" of the trainee. This is done by 
requiring answers to questions about the wiring system of the 
gun. Thus, the added objective may be stated as (5) identifying 
one or more probable causes of malfunction in the flow of current 
in the wiring system. 

The Sample Lesson 

The particular example employed in this lesson was the task 
of conducting a functional check of the M61A1 gun in the F-16 
aircraft. This procedure employs a checklist to test the several 
different voltages in the electric circuits that activate the 
gun. The purpose of the present s. iy was to seek evidence of 
the comprehensibility and workability of these directions for 
designing instruction in a number of Air Force personnel who were 
representative of potential users. These are typically airmen of 
intermediate rank who are well acquainted with the aircraft and 
its equipment, but are novices in instructional design methods. 

The lesson on the functional check follows this outline: 

Event 1. Gain Attention. The gun is named, and a graphic 
picture is displayed. A future development would use a picture 
with motion. 

Event 2. Describe the Goal. A descriptive text, with 
accompanying pictures, describes the process of making the 
functional check. Included are the purposes of the check, the 
locations of switches and connectors, rules for gaining access 
to the gun and its components, rules for safety. 

Event 3. Stimulate the Recall of Prior Learning. Messages of 
text remind the learner of previously learned information, safety 
rules, etc. 

Event 4. Present Material to be Learned. A picture is to be 
shown of the initial page of the checklist. 

Event 5. Provide Learning Guidance. This takes the form of 
displaying the direction for each step in the procedure, 
including expanded abbreviations, and an accompanying picture. 

Event 6. Check Performance. The learner is asked to carry out 
the performance of the checklist. Also, questions are posed 
relating to the flow of current in the electric circuits. 

Event 7. Provide Feedback. Corrective feedback is given as the 
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steps of the checklist are perf rmed. Feedback in the form of 
explanations are given to each ohoice made to multiple-choice 
questions, testing knowledge of current flow in the electric 
circuits. 

Event 8. Conduct Assessment. The suggestion is made that 
additional practice be given on the checklist procedure. 

Event 9. Enhance Retention and Transfer. For this checklist 
procedure, this event is considered to be adequately covered by 
events 6 through 8. 

The novice designer, in responding to these events, was 
asked to write out a script that described the CB instruction to 
be designed. It was f>xpected that such a script could be used by 
a computer programmer to devise the program of the lesson. A 
revised version of GAIDA is planned, which would enable a 
designer to display the contents of the nine events directly on a 
computer screen. This aspect of GAIDA design could not be 
included in the current formative evaluation study. 

Formative Evaluation 

According to Dick and Carey (1991), formative evaluation is 
conducted during the time in which instruction is being 
developed for the purpose of identifying strengths and weaknesses 
in the instruction and the need for revision. An essential early 
move in a formative evaluation study consists of "one-to-one 
trials", carried out with representative learners from the target 
group for whom the instruction is intended. These trials provide 
a look at the viability of the instructional linkings of content, 
setting, and learners. The three main criteria are considered to 
be as follows: 

1. Clarity. Are the directions clear? 

2. Impact. What is the effect of the instruction on 

the achievement of its objectives? 

3. Feasibility. Given certain support and time 

allocations, how feasible is the 
instruction? 

The formative evaluation study was conducted at Lowry Air 
Force Base. Six individuals participated in the study, each 
serving as a novice instructional designer. Five of these people 
were noncommissioned airmen who had experience as instructors in 
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the Armament Specialist course. One was a civilian who was an 
instructional developer in various aircraft maintenance 
specialties. All of these participants were acquainted with the 
F-16 aircraft and its equipment, including the M61A1 gun. In the 
jargon of evaluation studies, they were considered to be SMEs 
(subject-matter experts) . These men ran through the instruction 
individually, one in the morning and one in the afternoon, on 
each of three days. 

Each "designer student" (hereafter called the student) was 
seated facing the computer monitor screen, which rested on a 
table in front of him. He was told the purpose of his 
participation as a try-out of a computer-based lesson on 
designing instruction, containing an example of the checklist 
procedure for the M61A1 gun in the F-16 aircraft. He was told 
the instructions would be given on the screen. The investigator 
(Gagne) would be seated at his back, and would be available for 
questions if there were any. Also, he would be alert for any 
hang-ups with equipment operation, and would help out if called 
upon. The student was to describe the lesson being designed by 
writing its description (in the manner of a "script") in a 
notebook provided. i^»/ailable to the student were (1) a copy of 
the 32-step checklist, (2) a copy of the wiring diagram for the 
electrical circuits of the gun, and (3) a set of black-white 
drawings of the various components of the electrical system of 
the gun (switches, connectors, display panels, etc.). The 
checklist, of course, was an essential feature of the material to 
be taught. The wiring diagram was intended to provide a 
conceptual base for understanding the wiring system and its 
checking. The drawings of equipment components were to be used 
in assessing performance, after the check had been gone through 
at least once. 

Each student was asked to describe his designed instruction, 
following the Nine Events as an organizing principle. In 
addition, each student was asked to "think aloud" concerning the 
three criteria of evaluation previously mentioned (clarity, 
impact, feasibility) . Students wrote out the verbal 
communications they wished to make for each event, and also 
selected a drawn picture of a relevant switch, connector, or 
component, identifying it by letter from an array mounted on a 
display board. Students' oral comment, were recorded by means of 
a tape recorder. 

Duration of these sessions was approximately two hours, and 
was not recorded. At the close, each student was asked to state 
any general comments considered relevant, and helpful to the 
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design and revision of the CBI. 

General Findings 

The following trends and generalities may be summ&rized from 
the preceding protocols of six individuals: 

(1) Some errors of omission and transcription were noted by 
the students. These are correctable, and the revisions will lead 
to an improved lesson. 

(2) It would appear to be essential to provide designer 
students with information about the status of knowledge in the 
trainees for whom the training is intended. In the case of this 
lesson, a functional check of the M61A1 gun, designers need to 
know that trainees are assumed to have prior knowledge about the 
conformation of the aircraft, the location and function of its 
main components, and fundamental safety precautions. 

(3) Access to visuals as a component of instruction was 
given strong emphasis by these designer students. They tried to 
find and use the most detailed and realistic visuals that were 
possible to obtain. 

(4) The description of the goril 'Event 2), as presented in 
chis lesson, appeared to the students to be highly compressed. 

To be most effective, the description of the goal needs to have a 
more articulated organization than it has in this lesson. 

(5) The treatment of learning guidance (Event 5) in this 
lesson was considered appropriate by all the students. 
Essentially, this treatment consisted of: (a) presentation of the 
text of each step in the form of the checklist; (b) statement of 
the step directions in "plain English", by expansion of 
abbreviations; (c) a graphic presentation for each step that 
illustrated the particular equipment part and its location. 

(6) Students approved of Event 6, Checking Performance, and 
Event 7, Providing Feedback. They were able to suggest subject- 
matter for four multiple-choice questions, relating to checklist 
steps 6, 15, 16, and 20. The investigator did not require them 
to spend time on the precise formulation of questions to accord 
with acceptable psychometric principles. 

(7) Students agreed that an unprompted execution of e 32- 
step procedure was desirable for purposes of assessment. Scoring 
of performance could be done by checking off the steps, noting 
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where errors occurred. 

(8) It was thought that transfer (Event 9) could best be 
identified in terms of performance of the procedure, with 
checklist, on the aircraft. Several other possibilities exist, 
including the gun system in other models of the F-16 aircraft and 
in the F-15 aircraft. Some students stated that using a 
checklist to make a functional check would probably transfer to 
other procedures, such as a check of the bomb release system. 

(9) These students had no difficulties in comprehending 
instructional design processes, or in using them to devise a 
storyboard for a computer-based lesson on the functional check of 
the gun. While potential improvements such as pictorial 
additions were noted, students judged the lessons to be 
satisfactory and potentially effective. The GAIDA approach to 
teaching instructional design using a narrowly focused model 
appears to be feasible and capable. 

(10) Students experienced no difficulties in comprehending 
the instructions, and no difficulties in using them to design a 
lesson on the functional check of the M61A1 gun. Students 
considered the resulting lessons to be satisfactory for the 
instruction of trainees in the Armament Specialist field. 
Possible improvements by the addition of high quality visuals 
were noted. 

(11) It appears that this type of narrowly focused 
instruction, following the model for the particular task to be 
t-'.ught, is capable of mediating the production of reasonably good 
instruction. The resulting lesson design is produced with a 
small expenditure of time and effort. The question of transfer 
of learning from such an exercise to design for other varieties 
of task remains for future investigation. 



Conclusions 

It appears that the techniques of artificial intelligence do 
have legitimate applications in the domain of instructional 
systems development. We have described two research projects at 
the Armstrong Laboratory which incorporate some AI techniques. 
Other projects involving the application of AI to ISD exist and 
have met with some success (Duchastel, 1990) . 

There are a variety of interesting and worthwhile research 
projects to conduct in this area. For example, AIDA can be used 
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to systematically vary specific instructional parameters (e.g., 
sequencing of events, placements of objects on the screen, time 
allotted for learning activities, etc.) in order to determine 
optimal default settings for a range of instructional settings 
(different learning objectives, learner profiles, delivery media, 
etc.). GAIDA can be used to determine what transfer of learning 
exists between the sample lesson development task and actual 
lesson development activities. Transfer of learning with regard 
to such complex enterprises as courseware development is a 
largely unexplored domain. 

We have not devoted much attention to intelligent 
applications in the other four phases of ISD, but there is 
clearly a great deal of work being done in most of those areas as 
wel"" , The exception is that very little use of AI has occurred 
in Lhe maintenance phase. As a consequence, we urge exploration 
of AI applications in the maintenance phase. 

Finally, since we are largely ignorant of the details of 
human intelligence, we believe there is little to be gained in 
debating whether and how machines exhibit intelligence. Rather, 
we should focus our attention on the useful tasks that machines 
can be made to perform. If we are thoughtful in our efforts, we 
may succeed in making machines that contribute to learning. 
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